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Abstract: Wc introduce a Bernstein-type inequality which serves to uni- 
formly control quadratic forms of gaussian variables. The latter can for 
example be used to derive sharp model selection criteria for linear estima- 
tion in linear regression and linear inverse problems via penalization, and 
we do not exclude that its scope of application can be made even broader. 

A Bernstein-type inequality for quadratic forms of gaussian variables 

The concentration phenomenon of stochastic processes around their mean is 
of key importance in statistical estimation by model selection for getting non- 
asymptotic bounds for some statistics. For example in model selection via pe- 
nalization, for devising sharp penalties and proving useful upper bounds for 
the risk of an estimator, one needs generally to control uniformly the statistic 
of the risk of an estimator by means of a sharp concentration inequality. This 
topic has received since recently (late nineties) a considerable interest among 
the statistical community above all further to the amazing series of works of 
Michel Talagrand which can be seen as the infinite dimensional analogue of the 
Bernstein's inequality (see in particular (7) for an overview and (8) for later 
advances). Their application in non-asymptotic model selection has first been 
discovered by Birge and Massart (e.g. (2)), then refined and popularized by the 
same authors (e.g. (3; 4)). For beautiful lectures on the topic, we refer the dear 
reader to (5; 6). 

In this small body of work, we establish a new Bcrstein-type inequality which 
serves to control (e.g. uniformly) quadratic forms of Gaussian variables and 
which happens to be useful for controlling, for example, uniformly the quadratic 
risk of a finite (or a countable) set of linear estimators in linear regression and 
linear inverse problems (see (1) for an application). In the remainder, we will 
give both the uncorrelated form and the correlated form of such an inequality 
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Lemma 0.1. Let a = (a,k)k=i,p and b = (bk)k=i.p be two p— dimensional real 
vectors, and consider the following random expression : T = Y%=i a ^ z \ + °kZk, 
where Zk, k = 1, • • • ,p are i.i.d. N(0, 1), and let's put : a + = sup{sup fc=1 ... p {cifc}, 
a~ = sup{sup fc=1 ... p {— dfc}, 0}. Then the following two concentration results 
hold true for all x > : 



fc=i 



fc=i 



^2 a t + -ttVx + 2a + x < exp[ 



(0.1) 



fc=i 



- 2, 



bl 



y^a| + yVS - 2a x < exp[-x] (0.2) 
\ fe=i 



The proof of the lemma is rather technical, so it is deferred to the appendix 
section. 

The following lemma uses the concentration results of lemma (0.1) to control 
more general quadratic forms of Gaussian variables involving a matrix. 

Lemma 0.2. Consider the random expression T = z T Az + b T z, where A is 
p by p real square matrix, b is a p— dimensional real vector, and z = (zk)k=i,p 
is a p— dimensional standard gaussian vector, i.e. z k ,k = l,p are i.i.d. zero- 
mean gaussian variables with standard deviation 1. Let's denote by Sk,k = 
l,p the eigen values of the symmetric matrix ^(A + A T ^j, and let's put s + = 
sup{sup fc=1 ... p {sk},0}, and s~ = sup{sup fe=1 ... _{— Sfe},0}. Then, the follow- 
ing two concentration results hold true for all x > 



T > tr(A) + 2^\\A + A T \\2 + ^\\b\\^ + 



2s + x 



< exp[— x] 



(0.3) 



T<tr(A)-2 x /-\\A 



A T \\ 2 + -\\b\\ 2 Vx~-2 S -x <cxp[- 



(0.4) 



Proof. One can rewrite T as follows: T = z 7 



1 -,T 



(A + A^z + b 7 



and by using the eigen value decomposition of the symmetric matrix ^(A + A T ) 

one derives T = Ylk=i s k z 'k + b'^z', where s/., k = l,p are the respective eigen 
values of \ {A + A T ) , z' = U T z with U standing for the (orthonormal) eigen 
matrix of h (A + A T ) , and b' = U T b. Then, by noticing that z 1 stands for a 
p— dimensional standard gaussian vector, ||£>'|| 2 = ||6|| 2 , SjUi s k = t r {A), an d 



T,Usl = l\\A + A^\ 
follows immediately. 



so by applying lemma (0.1), the proof of lemma (0.2) 
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Appendix A: Proof of Lemma (0.1) 

Proof. We make use of the following lemma for proving lemma (0.1) 

Lemma A.l (Birge & Massart 1998). If a random variable £ satisfies for some 
two real positive numbers u and v the following inequality : 



logNE exp[y£] 



<P^, for all 0<y<- 
1 — vy v 



then 



£ > 2u^x - 



vx 



< exp[— x],for all x > 



(A.l) 
(A.2) 



We refer the reader to (2) for a proof of this lemma. 



Now, to prove lemma (0.1), one can notice first that concentration inequality 
(0.2) can be obtained from (0.1) by considering the random quantity 

T' = -T = J2(-a k )z 2 k + (-b k )z k 

k=l 

and by applying (0.3) on T' instead of T. So, we need to prove only (0.3). To 
do this, let us rewrite T as follows: T = J2k=i Tk, where T k = a k z\ + bkZk, and 
let us compute log [E(exp(y(T - T)))] , where f = YX=i ^ e nave 



E[exp(yT fe )] 



cxp 



-((1 - 2a k y)t 2 - 2yb k t) 



dt 
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M[exp[yT k }] = exp 



1 - 2a k y 



exp 



1 - 2a k yt- 



bkV 



E[exp[yT k }] = 





r b i y * 1 

2 V 


exp 


l~2a k y 



E 



exp[y(T fc - Ofc)] 
log (E[exp[j/(T fe -a fc )] 



exp 



l-2a fc y 



exp[-ya fc ] 



t^ 2 



V 1 - 2a fe2/ 



1 - 2a fc y 2 

Then, by putting a + = sup { sup ft=1 ... , p {a k }, 0}, one derives (see the technical 
details below) that for all < y < j^f 



log E 



cxp[?/(T fe - Ok)] 



< (a 2 + ^ 2 < (al + ^)y 2 



1 - 2a k y ~ 1 - 2a+y 
which implies by independence that for all < y < 



log 



P <al+ b -±W 



exp[ y (T-T)]j)<g^™ 



log 



exp[2/(r-f)] 



< 



ELi (" 2 fc + f)> 2 

1 - 2a+y 



Finally, by applying lemma (A. 2) below with u = \JJ2k=i( a k + ~t) > an< ^ 
2a + , one derives that for all x > 

p 

P T> [J^Ofc] +2 



fe=i 



65 



\ I] ( a fe + 2a+x J ^ exp[-a;] 

\ fe=i 



This terminates the proof of lemma (0.1) 

Some additional technical details about the proof 

We will show here that for all r > 0, a > r, and < y < one has 

— 1 r 2 w 2 

— log(l - 2ry) -ry < — 

2 1 — 2ay 



□ 



(A.3) 
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and that for all r < 0, for all a > 0, and for all < y < j-, one has 

1 T 2 2 

- - log(l - 2ry) -ry< (A.4) 

Proof, let us start by showing inequality (A. 3). To do this, let us consider the 
following function 

1 r 2 y 2 



fr, a (y) = - o lo s( 1 ~ 2r y) - n/ - 



2 v ' l-2ay 

One first notices that f r ,a(0) = 0, then a sufficient condition for inequality (A. 3) 
to hold true is that f r , a {y)' < 0, for all < y < j-. We have 

/ ri0 („) = :ilog(l-2ry)-n,+ ^+ ^ ^ 



2 ov a/ 2a (2a) 2 \-2ay 

one then derives that 



fr,a(y)' = --, "Ti r 



fr,a(y)' 



\-2ry ' 2a (1 - 2ay) 2 
2r 2 y r 2 y r 2 y 



1 - 2ry (1 - 2ay) (1 - 2ay) 2 
2r 2 y 2r 2 y 



fr,a(y)'< 1 _ 2ry {1 _ 2ay) 

and finally since j-. — ^ — r < j-, — ^ — r, one deduces that 

•> (l-2ry) — (l-2ay) ' 

2r 2 y 2r 2 y 



frAy) ' " (1 - 2ay) ( 



1 - 2ay) 



= 



then we have shown (A. 3). 

We proceed in the same way as for showing inequality (A. 3) to show inequality 
(A.4). So let us consider the following function 

— 1 r 2 y 2 

9r,a{y) = -7T ~ 2r V) ~ r V ~ 



2 v ' l-2ay 

One first notices that g r , a (0) = 0, then a sufficient condition for inequality (A.4) 
to hold true is to that g r ,a(y)' < for all < y < One derives that 

2 r 2 

grAv) = - — r 



9r,a(y)' = 



\-2ry ' 2a (1 - 2ay) 2 
2r 2 y r 2 y r 2 y 



1 - 2ry (1 - 2ay) (1 - 2ay) 2 
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2r 2 y 2r 2 y 



9r,a(y)' ^ 



1 - 2ry (1 - 2ay) 



and finally, since l J 2r — (i-\a ) > one nnc ^ s that 



9r,a{y)' ^ 



2r 2 y 



2r 2 y 



(1 - 2ay) (1 - 2ay) 



□ 



Birge's & Massart concentration inequality 

Lemma A. 2. If a random variable £ satisfies for some two real positive numbers 
u and v the following inequality : 



log \E exp[y£] 



<MLj oraU o <y< - 

1 — vy v 



then 



IP £ > 2u\fx + vx < exp[—x],for all x > 
The proof of this lemma can be found in (2). 



(A.5) 
(A.6) 
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